Data Integration

Data Integration is the process of combining data from multiple sources like social media, IOT devices, customer transactions, data warehouses and so on. You perform append or overwrite operations on this combined data and create data partitions to optimize data querying. This data is then loaded into a data lake.

Calibo's Data Pipeline Studio (DPS) provides templatized integration jobs with around 30+ templates using various supported combinations of data sources, integration tools and data lakes. Apart from templatized data integration, DPS also supports custom integration jobs in which you can use custom code. After the data integration job is run and the data is loaded in a data lake, it can be used for further processing like transformation, data quality, data visualization and so on.

The pipeline for a typical data integration job consists of the following stages: Data Source > Data Integration > Data Lake

Data Integration Job pipeline

The Lazsa Platform provides support for various technologies in data integration jobs in a data ingestion pipeline.

Type of integration job	Supported technologies
Templatized data integration	Data Integration using Databricks Data Ingestion from SFTP to Snowflake Data Ingestion using Amazon Kinesis Data Streams with S3 Data Lake Data Ingestion using Amazon Kinesis Data Streams with Snowflake Data Lake Data Integration using Amazon AppFlow Data Integration using Snowflake Bulk Ingest Data Integration using Snowflake Stream Ingest

What's next?Data Ingestion from SFTP to Snowflake